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Noting that use of the reading-related components 6f 
the Natit>nal Assessment of Educational l^Vogress (NAEP) by state 
education agencies has ranged 'from extensive to moderate to limited, 
this paper presents case studies of the ways in Which states have 
Ctsed the NAEP models. The first half of the pi^per descrdbes extensive 
use by Minnesota and Connecticut, moderate use by the state of Maine, 
and limited use by Wyoming and North Dakota. The second half of the 
paper discusses future uses of the NAEP methodology and materials and 
list's gug'stions that state education agencies might want to ask when 
considering adaptation of NAEP reading components. (HTH) 
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USES OF NAEP I!* READ;!*G BY STAT^, EDUCATION AGENCIES 
In the 1970's and earlyri980' a use of the reading-related components 
of the National Assessnigfttt of Educational Progress by state education 
agencies (referred to in most states as "the State Department oTlEducation'') 

has ranged from eictensive to moderate^ or to limited utilization . Sebring, 

. ' ' ' ' ^ • V 

in 1981 and 1982, conducted a series of case studies pertaining to how 

>.' 

state, education 'Agencies haVi used NAEP in any and all ways. ^Among her 

findings specifically related to reading were the following; 

Jr Examples of Extensive Utilization 

M innesota , The Minnesota Statewide Assessment Program has relied 
heavily on the NAEP model since the early 1970' s. Compreherfoive 
assessments in readirtg were made in 73-74, 80-81 . (secondary grades), 
and 8J-82 using intact classroom samples. Comparisons were made 
between Minnesota students,^ Central' Region students, and natioAal 
I sample students on the items that were taken directly from NAEP//* 

A report was then presented t*b the State Board of Education. It 
thereby beca/ne an official part of the public record, resulting in 
extensive radio , TV, and news^paper coverage , 

Department of Education personnel Responsible for readinjg provided 
assistance to a local IRA council which produced materials j^nd 
workshops aimed at. improving achievement in those areas where student 
performance was low, A second major* utilis^^tion was realized through 
the staters Piggyback Program, which enabled local distficts to 
assess their students with statewide a8ses$ment measures. In 1^74, 
3% of Minnesota's di8j:ricts participated in the Piggyback Program, 
while in 1980-81, 56% were involved. Evaluations of these two 
uses Indicated that recommendations and materials steimning from 
assessment da<ta had an impact on. classroom instruction, \A third 

* utilization of assessment findings was in the form of information 
the state Director of the Assessment Section frequently offered 
to legislative committees and individual legislators'. However, 
in Minnesota no statewide decisions wejre made by the legislature 

' because the philosophy of the state faVojs declsion**making at 
the local school level. 

Conn ecticut . Extensive use of NAEP reading material was made in 
Connecticut since the Connecticut Assessment ot Educational Pro-,, 
• " greas was modeled after NAEP in *terms of its basic .goals, design, 
and implementation. Beading was assessed irt 71''72^ 75-76, 77*-78 
and 78-79 in o«*der to make s^dte, Northeast regional, ftnd national 
comparisons^ . ' * 



'/^fter each assessment, reports were developed and sent in brochu 
• form to legislatoriB , parents, and local hoards 'of education and 
in more detailed form to teachers, administrations, and public 
libraries. Interpretation workshops were al$o presented to 
representatives of approicimately 100 communities.. 

Fifty-five districts *opted to have all or a sample of their 
studeht-s tested in reading and compared agaiiist themselves 
and/or statewide and national samples on the 1975-76 reading 
assessment. Forty-four districts used this option with the 
1978-79 reading assessment. Assessment tiata was held confi- 
dential to the districts, but the Connecticut Department of 
Education sponsored meetings at which district data were inter- 
preted without violating, confidentiality. District cost 
amounted to $1.95 per student tested. 

Statewide assessment results were used to formulate recommen- 
dations related to reading areas needing improvement. In 
1980, reading consultants produceo^a i^ooklet for teachers con- 
taining suggestions for ''ge1:tirtg main ideas." This aid was 
distributed to all schools in ^Connecticut . 

In 1981, work on the State's Ten-Yeat Plan wa$ in progress. 
The Connecticut Assessment of Educational Progresft provided 
logitudirial and comparative data and national and| state 
comparisons in reading (and 10 other subjects) so that local 
schools and the state would be able to coopera^te in pjLanning, 
impalement ing, and evaluating school programs.. Since local 
control of schools is strong th<? Connecticut, recommendations 
emerging from the assessments and state plans will be imple- 
mented at district levels. 

Exam ple of Moderat e Util i :^ation 

Maine . After two panels of Maine educat^Ors reviewed NAEP objec- 
tives, the .^AEP model was selected to Assist the state in its 
yari6us assessments. Nine and seventeen year-olds were assessed 
in reading in 1973 and 1974, respectively . Forty-seven of the 
75 itemsr used to assess reading in 197A were taken from NAEP, 
with tj;/e^ remaining items developed especially for the state. 
Sampl/ing. procedures paralleled NAEP techniques. Reading itemp 
wer^ placed in three packages and then given to three sep^ifate 
samples of students. State department personnel w^e- trained 
to administer the reading (and writing) ^tests and used {jaced 
audio tap^ to givfek test directions. 

Reports were developed explaining the meaning ot significant 
differences among the test performance of Maine ^ regional, 
and national samples. Achievement was also summarized for 
various groups in terms of such considerations as socio-- 
economic stati^, levels of parental education, community 
sisse, and so on* 1^ wafi pt>inted out that Maine educators 
'needed to decide the practical significance of any differ- 
enced that were noted, though. In order to establish 
frameworks for evaluating performance in reading/ over 260 



teachers were randomly selected to judge minimal, desired, 
and predicted oiftcomes for 15 reading items aod the diffi- • 
culty and appropriateness of six reading passages. A Reading 
Ihterpiretation Committee (comprised of 16 elementary, setron- 
dary, and university teachers) was then formed and produced 
a' report detailing interpretations and recommendations for 
educators working af local school , state department, and 
collegiate levels^ in Maine, • 

• * " ." 

Other reports of varying coftiplexity were sent ^o district 
superintendents, principals, and curriculum supervisors. 
Three thousand copies of the reading (an<^f^Viting) assess- 
ment pf 9 year-olds conducted in 197A wfre sent out; articles 
appeared in newspapers of the state; a series of articles 
appeared in the Maine ffeacher . Finally, the state education 
agency in Maine issued a^ report noting that the ljil/A» reading 
assessment resulted in a district level in-servicb program 
related to reading and writing and a state department of 
education devejroped approach to the teaching of basic skills. ^ 

For the 1977-78 SLtatewide assessment of basic skills, 38 NAEP 
reading items were used for the eighth grade yest, *while 40 were 
incorporated the • eleventh -grade test.- A citMen's committee 
was formed to work together with the Maine »Mt% department 
in writing such, interpretations^ and recommenoations as the 
following: the legislature should not mandate minimum competency 
requirements for graduation; the state should provide technical 
assistance to scliools related to establishing performance stand- 
.ards, to using, and to interpreting assessment data; the state 
should offer districts an assessment model which they could 
choose to adopt' or modify.- 

Thus, in Maine, the emphasis in the early 1^80*s was upon evalua- . 
tlon at the district level, tite state department of education's 
role was to provide leadership and technical' asl^istance as districts 
carried out their own assessments of achievement at not only minimum, 
bu^ also maximum, that is, excellence levels. The only requirement 
was that the public be inforrtied as to Jtudent progress toward 
expected learning outcomes. In this regard districts were free 
to decide whether they wishe'4 to use NAEP objectives and items 
and could call upon appropriate state department o'f education 
staff for technical help. ^ 

Examples of Limi»1,ed Utili^feation • 

' • . <» ■ 

Wyoming . In 1971 individuals from th©^ College of Education at , 
the University of Wyoming, the state Departfment of Education, 
t^ie ESiEA State Advisory Committee, the Wyoming Education Asso- 
ciation, 20 professional educatiVfi' associations , and the general 
publl^^established educational goals and objectives. A*fter 
items from a wide ♦^riety of sources, including NAEP, were 
reviewed, an assessment in reading and four other areas wa^ 
condut:ted involving high school seniors. ' 

However, although NAEP item developm^t procedures and some NAEP 
itemi^ V^t^ empl*oyed, assessment results were ^ot disseminated 



.broadly. State department personnel were able to make assessment 
information available to teachers at the district level, but 
could, not mandate use of the information because dictates .of r 
this nature, would violate local control of schoolp. 

North Dakota , In 1976-77, NAEpCbject ives, iotems, and dal^a^were 
used in a statewide reading assessment of fourth, eighth, and , / 
eleventih grade students. Approximately 1400 youngsters at each 
• of these grade levels are tested by means of a stratified (on 

the basis of school enrollment) random , sampling procedUre, 
Word identification and recognition, word and sentence compre- 

. hension, comprehension of longer discourse, and reading study 
skills items were selected from NAEP i^leased exercises and * 
the Minnesota^ assessment and then were administered by district 
staffs. The results were noted in term^ of correct answers 
related to each objective and subobjective' and compTarisons were ^ 
reported between state students. Central Region students, and 

' ^the national NAEP sample. Findings led to various recommepda- 
tions in state department repcXrts (e,g,, that content area reading 
should be emphasized, that districts "Use various assessment 
objectives dn reading to write objectives). Schools* pai^ticipating 
in the assessment received a short report showing how th^y stood 
up against statewide results, but comprehensive comparisons cbuld 
not be drawn because few schools had' decided to administer all 
of the reading assessment items; 

Finally, cohcise, readable reports of the assessment were sent 
^to administrators in each of the state's districts. The state 
Right to Read Coordinat9r included assessment ir^ormation into 
various workshops, for teachers, principals, and superinytjindents 
-conducted 1976-1980, Recommendati€>ns (distributed in booklet 
form)| emphasized the use of criterion referenced measures rating 
various aspects of existing reading programs to ascertain needed 
changes. Aside from these efforts, the state department, dye to 
limited resources , was not able^to disseminate and encourage 
widespread use of the assessment results in the schools of the 
state, ^ * • 

^ •- . 

Reports summarizing the assessment were also sent to the State 
Board of Education dnrf to the Legislature, showing hbw students 
in the state compared with the assessment performance of students 
nationwide^. Since state funds account for more than 60% of local 
school district budgets , legislators viewed student"T?^formance on * 
criterion-referenced and norm- referenced medsi\res wi th riiterest , 

Certainly the use of «NA£P reading components by state education 

agencjl^^s cannot be defined on the basis of the experiences ^f five 

; N ' . ■ V 

stat^. LaPointe and Koffler (1982) have asserted their belief that, the 
general impact of NAEP has betfn less than strong, but Tyler (1982) hA^ 
challenged their assertion. Perhaps judgments in thife matteic can be made 



though, when the results of a December 1982 survey conductetl by the NAEP 
Utilization and Liaison Department are in. The survey was settt to all 
SEA' 8 an« contained questions related to T^ast ises of NAEP obiectives 

I ■ 

items, results,^ and methodology as. well as other queries related to. 
future -l^^potential uses- of NAHP materials and services. In the ques- 
tionnaire, new' plaiVa were descrijied pettaining to the following: 



/ 



1. new and continuing forms of general and technical 
- assistan<^e to SEA's; 

2. joint SEA/ NAEP development of objectives and itemsK 

3. concurrent SEA/NAEP assessment arrangement involving 
a. revised item release policy (SEA*s tould use .some or 
all NAEP items the same year they are used nationally 
and thereby make more timely and less expjensive com- , 
parisons) . 



Future Upes 



The experiences of the fiwe states described above demonstrate that 
SEA's have adapted ,or^adopted the reading aspects of NAEP methodology and 
materials in keeping with the more general cfonstraints and needs of their 
'respective circumstances. Involvement in the future will continue, to 
vary in light of political, fiscal, and "local control of schoolsJ' consijd- 
eratiorfsx ♦ ' 

* In terms of ''test consumer" concerns, SEA's may want 'to ask questidjis 

such as the following when they consider alignments with NAEP reading * 

« ■ ■ ■* 

components: • * . . 

Ex^amplc Questions Related to Purposes for Testin g 

. 1. . How can NABf help us analyze reader strengths ^nd weaknesses /§ 
in skills, attitudes, habits, and interests? ^ 

2. How cah NAEP help us develop reading curriculum guidelines in 
terms of our prioritised educat^ional ^and vocational needs' and 
in terms of both minimum competencies and st^andards of excellence? 

3! How*" can NAEP help us guide and place students in the K-12 
4|equence as well as in community college, college, or 
e^loyment sectors? i 



■/ ■ ■■• ■ • ■ ■ir' ■ , ' 

How can NAEP help us determit^e whether-^raading is declining', 
/Remaining stable, or improving? . . ^ 

How can NAEP help ue c^ndomly, sample students , average their 
performance, and come up with composite scores for the various 
aspects of maturity in reading? ^ . - 4 

How can NAEP help us provide data to legLelative committees 

so that we can build recommendations for additional funds for 

certain xareas o# need (e/g. , reading in "the middle school 
and at the secondary levels)? ^ 

How can NAEP help us determine, whether our remedial reading 
programs are effective? . . 

In what specific ways will NAEP data contribute to improved 
citizens! *and legislators ' understanding o-f reading achieve- 
ment? ^ > 

^\ * . 

In what specific ways will NAEP data and services enable' pur 
SEA to provide assistance In the area of reading at district 

and classroom levels? — ' • 

Example Questions Related to Validity and Reliability - 

Do N^P test items in general and in particular match our 
state's prioritized goals and\pbjectives of reading instruc- 
tion (instructional validity)? ^ 

How well will NAEP reading items correlate with our students* 
scores on other tests, with school grades, with future grades, 
and'' with readin^g performance in nonschool settings\(statistical 
validity)? , ? 

Is the contenttfcof the NAEP reading section approved by "dis- 
interested/' that is, unbiasefd, external reviewers, including 
testing experts, teading specialists, researchers , "administrators 
teachers, school board members, representatives from business and 
labor, and students themselves? How were these external reviews 
obtained? . ' , i 

What useful comparisons will NAEP items provide the SEA personnel 

Why should state; department personnel select NAiSP items, 
procedures, and results . instead of or in addition to items 
contained in norm-referdpficed tests used and respected' 
nationwide?j^ 

How have NAERi terns been reviewed for sex, ethnic, geographical^ 
and other foriM of bias? — . ^ 

In what ways are NAE? items and results, demonstjrated reliable 
and consistent? ^ : 



Exaihples of Questions Related to Cost Ef^fectiveness 

^ ■ . ' ' • • • .* . ■ 

1^. How rejjisable are NAEP materials? 

• • 2> How expensive (in terms of . time and money) are scoring (^,g. , 

""achine) and reporting (e.g., printout) aspects? 

y * 3- ^at are "hidden costs'' for time and training ofr U^ers, 

administratoys , and' interpreters? 

4. What interpretive aids (A-V, booklets, etc.) and services 
does JJAEP Offer? - . 

*■ - 

" , ^* scores reported injj|anners that will make them ^ 

directly usable in decisipn-raakSg, processes? 

■' 6. Do the costs (apparent as well as hidden) pf utilizing NAEP 

items, procedures', and results justify the use of these in 
\ight o{ benefits to the state? 

. 7. How can NAEP help us economize out statewide reading .assessment ' 

procedures by using accurate,;, stratified random sample selection 
technique?. - . 

8.- How can NAEP help us streamline our state assessment records 

and- procedures so that we are not collecting diata inefficiently 
and at unnecessarily high expense. , ♦ • 
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